Skip to content

Avoid f64 precision loss when parsing integer byte counts#171

Open
sayantanmandal1 wants to merge 1 commit into
bytesize-rs:masterfrom
sayantanmandal1:fix-parse-precision
Open

Avoid f64 precision loss when parsing integer byte counts#171
sayantanmandal1 wants to merge 1 commit into
bytesize-rs:masterfrom
sayantanmandal1:fix-parse-precision

Conversation

@sayantanmandal1

Copy link
Copy Markdown

Parsing a byte count with a unit suffix loses precision for large integers,
because the suffixed path in from_str always parses the number as f64:

"9007199254740993".parse::<ByteSize>()   // 9007199254740993
"9007199254740993B".parse::<ByteSize>()  // 9007199254740992  <- off by one

f64 only has 53 bits of mantissa, so anything at or above 2^53 gets rounded once
a unit is involved. The no-suffix path already parses straight to u64, so it
stays exact.

When the number has no decimal point this parses it as u64 and multiplies by the
unit factor directly. Decimal inputs like "1.5GiB" still go through f64, and
overflow still saturates like before. Added a test for 2^53 + 1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant